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Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD)  has 
stated  that  the  development  and 
implementation  of  DEAMS  is  critical  to 
the  department’s  goal  of  producing 
auditable  financial  statements  by 
September  2017.  In  October  2010, 
GAO  reported  that  although  the  Air 
Force  had  developed  a  cost  estimate 
that  met  best  practices,  it  had  not 
developed  a  schedule  that  met  best 
practices  for  implementing  DEAMS. 
GAO  has  published  guides  that  identify 
the  characteristics  and  associated  best 
practices  for  developing  reliable 
schedule  and  cost  estimates.  GAO 
was  asked  to  review  the  schedule  and 
cost  estimates  for  selected  DOD 
systems.  This  report  addresses  the 
extent  to  which  the  current  schedule 
and  cost  estimates  for  DEAMS  were 
prepared  in  accordance  with  GAO’s 
Schedule  and  Cost  Guides. 

Specifically,  GAO’s  review  focused  on 
the  schedule  and  cost  estimates  that 
supported  DOD’s  February  2012 
Milestone  B  decision,  which 
determined  that  investment  in  DEAMS 
was  justified.  GAO  assessed  the 
schedule  and  cost  estimates  and 
supporting  documentation.  GAO  also 
assessed  an  updated  schedule  dated 
October  2012.  GAO  interviewed 
DEAMS  program  officials,  lead 
schedulers,  and  cost  estimators. 

What  GAO  Recommends 

GAO  recommends  that  the  Secretary 
of  the  Air  Force  update  the  cost 
estimate  as  necessary  after 
implementing  GAO’s  prior 
recommendation  to  adopt  scheduling 
best  practices.  DOD  concurred  with  the 
recommendation. 


View  GAO-14-152.  For  more  information, 
contact  Asif  A.  Khan  at  (202)  512-9869  or 
khana@gao.gov  and  Nabajyoti  Barkakati  at 
(202)  512-4499  or  barkakatin@gao.gov. 
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DOD  BUSINESS  SYSTEMS  MODERNIZATION 

Air  Force  Business  System  Schedule  and  Cost 
Estimates 


What  GAO  Found 

The  Air  Force’s  schedule  that  supported  the  February  2012  Milestone  B  decision 
for  the  Defense  Enterprise  Accounting  and  Management  System  (DEAMS)  did 
not  meet  best  practices.  The  cost  estimate  did  meet  best  practices,  but  the 
issues  associated  with  the  schedule  could  negatively  affect  the  cost  estimate. 
GAO  found  that  the  schedule  supporting  the  Air  Force’s  decision  to  invest  in 
DEAMS  partially  or  minimally  met  the  four  characteristics  for  developing  a  high- 
quality  and  reliable  schedule.  For  example,  the  schedule  did  not  reflect  all 
government  and  contractor  activities,  and  resources  were  not  assigned  to 
specific  activities.  The  schedule  also  lacked  a  valid  critical  path,  preventing 
management  from  focusing  on  the  activities  most  likely  to  cause  critical  program 
delays  if  they  are  not  completed  as  planned.  In  addition,  a  schedule  risk  analysis 
was  not  conducted  to  predict  a  level  of  confidence  in  meeting  the  program’s 
completion  date. 


Extent  DEAMS  Schedule  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Partially  met 

Well-constructed 

Partially  met 

Credible 

Minimally  met 

Controlled 

Partially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Air  Force. 


GAO  found  that  the  October  2012  updated  schedule  estimate  was  not 
comprehensive,  well-constructed,  and  credible,  and  contained  weaknesses 
similar  to  those  found  in  the  previous  schedule.  In  May  2013,  program  officials 
provided  a  third  schedule  that  they  said  contained  some  improvements  but 
acknowledged  that  issues  remained  that  prevented  the  schedule  from  meeting 
best  practices. 

GAO  found  that  the  DEAMS  cost  estimate  fully  or  substantially  met  the  four 
characteristics  of  a  high-quality  and  reliable  cost  estimate.  For  example,  the  cost 
estimate  included  both  government  and  contractor  costs  for  the  program  over  its 
life  cycle  and  provided  for  an  independent  assessment  and  reconciliation. 
Because  the  cost  estimate  relies  on  dates  derived  from  the  schedule  and  GAO  is 
questioning  the  reliability  of  the  schedule,  the  credibility  of  the  cost  estimate 
could  be  affected. 


Extent  DEAMS  Cost  Estimate  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Fully  met 

Well-documented 

Substantially  met 

Accurate 

Substantially  met 

Credible 

Substantially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Air  Force. 
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Abbreviations 


DEAMS  Defense  Enterprise  Accounting  and  Management 

System 

DOD  Department  of  Defense 

ERP  enterprise  resource  planning 
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GAO 

441  G  St.  N.W. 
Washington,  DC  20548 


U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


February  7,  2014 
Congressional  Requesters 

The  Department  of  Defense  (DOD)  invests  billions  of  dollars  to  develop 
and  implement  enterprise  resource  planning  (ERP)  systems,  which  it 
considers  critical  to  transforming  the  department’s  business  operations 
and  addressing  some  of  its  long-standing  weaknesses,  including  those 
related  to  financial  management  and  business  systems  modernization.1 
DOD  has  stated  that  the  development  and  implementation  of  the  Air 
Force’s  Defense  Enterprise  Accounting  and  Management  System 
(DEAMS)  is  critical  to  the  department’s  goal  of  producing  auditable 
financial  statements  by  September  201 7, 2  as  called  for  by  the  National 
Defense  Authorization  Act  for  Fiscal  Year  2010. 3 

In  October  2010,  we  reported  that  although  the  Air  Force  met  best 
practices  in  developing  a  cost  estimate,  it  did  not  meet  best  practices  in 
developing  the  schedule  estimate  for  implementing  DEAMS.4  Having 
such  a  schedule  is  crucial  to  the  Air  Force’s  ability  to  reliably  estimate  the 
program  completion  date. 

To  support  Congress’s  continuing  oversight  of  DOD’s  progress  in 
implementing  its  ERP  systems,  you  asked  us  to  review  the  schedule  and 
cost  estimates  for  selected  DOD  ERP  systems.  The  objective  of  this 
review  was  to  determine  the  extent  to  which  the  current  schedule  and 
cost  estimates  for  DEAMS  were  prepared  in  accordance  with  GAO’s 


ERP  system  is  an  automated  system  using  commercial  off-the-shelf  software 
consisting  of  multiple,  integrated  functional  modules  that  perform  a  variety  of  business- 
related  tasks,  such  as  general  ledger  accounting,  payroll,  and  supply  chain  management. 

2For  a  complete  list  of  the  ERP  systems  that  DOD  has  identified  as  essential  to 
transforming  its  business  operations,  see  GAO,  DOD  Financial  Management:  Reported 
Status  of  Department  of  Defense’s  Enterprise  Resource  Planning  Systems,  GAO-1 2-565R 
(Washington,  D.C.:  Mar.  30,  2012). 

3Pub.  L.  No.  111-84,  §  1003(a),  (b),  123  Stat.  2190,  2439-40  (Oct.  28,  2009). 

4GAO,  DOD  Business  Transformation:  Improved  Management  Oversight  of  Business 
System  Modernization  Efforts  Needed,  GAO-11-53  (Washington,  D.C.:  Oct.  7,  2010). 
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Schedule  and  Cost  Guides.5  We  reviewed  the  most  current  schedule  and 
cost  estimates  that  supported  DOD’s  February  2012  Milestone  B 
decision,  which  determined  that  investment  in  DEAMS  was  justified.6 


Scope  and 
Methodology 


We  assessed  the  DEAMS  schedule7  using  the  GAO  Schedule  Guide  to 
determine  whether  it  was  comprehensive,  well-constructed,  credible,  and 
controlled.8  To  assess  the  schedule,  we  obtained  and  reviewed 
documentation,  including  the  integrated  master  plan  and  work  breakdown 
structure.9  In  assessing  the  program’s  cost  estimate,  we  used  the  GAO 
Cost  Guide  to  evaluate  the  DEAMS  Program  Management  Office’s 
estimating  methodologies,  assumptions,  and  results  to  determine  whether 
the  cost  estimate  was  comprehensive,  well-documented,  accurate,  and 
credible.  We  obtained  and  reviewed  documentation,  including  the 
program  office  estimate,  software  cost  model,  independent  cost  estimate, 


5GAO,  GAO  Schedule  Assessment  Guide:  Best  Practices  for  Project  Schedules — 
Exposure  Draft,  GAO-12-120G  (Washington,  D.C.:  May  2012),  and  GAO  Cost  Estimating 
and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital  Program 
Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009).  The  guides  document  the  specific 
characteristics  and  associated  best  practices  for  preparing  schedule  and  cost  estimates. 

Milestone  B  is  considered  the  official  start  of  the  program  according  to  the  Defense 
Acquisition  Management  System  Framework.  The  framework  is  intended  to  translate 
mission  needs  and  requirements  into  stable,  affordable,  and  well-managed  acquisition 
programs. 

7Although  the  Air  Force  provided  us  with  its  integrated  master  schedule  for  the  DEAMS 
program,  we  determined  that  it  did  not  meet  the  characteristics  of  an  integrated  master 
schedule  in  accordance  with  GAO’s  Schedule  Guide.  Specifically,  an  integrated  master 
schedule  is  a  document  that  integrates  the  planned  work  by  the  government,  contractor, 
and  other  key  parties;  the  resources  necessary  to  accomplish  that  work;  and  the 
associated  budget.  Therefore,  for  purposes  of  this  report,  we  use  “schedule”  to  refer  to  the 
Air  Force’s  integrated  master  schedule. 

8A  schedule  is  controlled  if  it  is  updated  periodically  by  trained  schedulers  to  realistically 
forecast  dates  for  program  activities  and  compared  against  a  designated  baseline 
schedule  to  measure,  monitor,  and  report  the  project’s  progress. 

9An  integrated  master  plan  provides  an  event-based  hierarchy  of  program  events,  with 
each  event  supported  by  accomplishments  and  each  accomplishment  associated  with 
specific  criteria  to  be  satisfied  for  its  completion.  The  plan  is  normally  part  of  the  contract 
and  is  therefore  contractually  binding.  A  program’s  work  breakdown  structure  defines  in 
detail  the  work  necessary  to  accomplish  a  program’s  objectives,  including  activities  to  be 
performed  by  both  the  government  and  contractors. 
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and  risk  and  uncertainty  analysis.10  We  also  interviewed  key  program 
officials,  such  as  the  Program  Manager,  lead  schedulers,  and  cost 
estimators,  to  obtain  information,  such  as  explanations  to  resolve 
identified  discrepancies. 

After  we  briefed  DEAMS  program  officials  on  the  results  of  our 
assessment,  they  provided  an  updated  schedule  dated  October  2012.  For 
this  updated  schedule,  we  determined  the  extent  to  which  it  met  certain 
best  practices  for  the  comprehensive,  well-constructed,  and  credible 
characteristics,  because  not  implementing  these  best  practices  would 
affect  the  reliability  of  the  entire  schedule.  In  May  2013,  program 
management  officials  provided  another  updated  DEAMS  schedule,  which 
they  acknowledged  contained  issues  that  prevented  the  schedule  from 
meeting  best  practices.  Although  we  did  not  independently  assess  the 
May  2013  schedule,  we  did  confirm  that  it  included  certain  information 
needed  for  long-term  planning. 

We  conducted  this  performance  audit  from  May  2012  to  February  2014  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


DEAMS  was  initiated  in  August  2003  and  is  intended  to  provide  the  Air 
Force  with  the  entire  spectrum  of  financial  management  capabilities, 
including  collections,  commitments  and  obligations,  cost  accounting, 
general  ledger,  funds  control,  receipt  and  acceptance,  accounts  payable 
and  disbursement,  billing,  and  financial  reporting  for  the  general  fund.11  In 


10An  independent  cost  estimate  is  another  estimate  based  on  the  same  technical 
information  that  is  used  to  validate  and  cross-check  the  cost  estimate,  but  is  prepared  by  a 
person  or  organization  that  has  no  stake  in  the  approval  of  the  program.  A  risk  and 
uncertainty  analysis  recognizes  the  potential  for  error  and  attempts  to  quantify  it  by 
identifying  the  effects  of  changing  key  cost  drivers  (e.g.,  costs  associated  with  personnel 
or  software  development). 

^An  agency’s  general  fund  accounts  are  those  accounts  in  the  U.S.  Treasury  holding  all 
federal  money  administered  by  an  agency  that  is  not  allocated  by  law  to  any  other  fund 
account. 
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February  2012,  the  DOD  Deputy  Chief  Management  Officer  granted 
Milestone  B  approval  for  DEAMS  to  enter  the  Engineering  Development 
Phase  of  the  acquisition  life  cycle,  which  is  considered  the  official  start  of 
the  program.  DEAMS  program  functionality  is  intended  to  be  implemented 
across  the  Air  Force  in  a  series  of  releases  in  two  increments — Increment 
1  will  include  six  releases  and  Increment  2  will  include  two  releases.12 
DOD  has  approved  the  funding  for  the  Air  Force  to  proceed  with  the 
acquisition  of  the  functionality  for  the  first  increment  of  DEAMS.  This 
funding  is  approximately  $1.6  billion,  with  deployment  scheduled  to  occur 
during  the  fourth  quarter  of  fiscal  year  2016.  The  Air  Force  reported  that  it 
had  spent  about  $427.5  million  as  of  September  30,  201 3,  on  the 
program. 

As  stated  earlier,  in  October  2010,  we  reported  that  although  the  Air 
Force  met  best  practices  in  developing  a  cost  estimate,  it  did  not  meet 
best  practices  in  developing  the  schedule  estimate  for  implementing 
DEAMS.13  In  particular,  the  Air  Force  had  not  developed  a  fully  integrated 
master  schedule  that  reflected  all  government  and  contractor  activities. 
We  recommended  that  the  Air  Force  develop  an  integrated  master 
schedule  that  fully  incorporated  best  practices,  such  as  capturing  all 
activities,  sequencing  all  activities,  integrating  activities  horizontally  and 
vertically,14  establishing  the  critical  path  for  all  activities,  identifying  float 
between  activities,15  conducting  a  schedule  risk  analysis,  and  updating 
the  schedule  using  logic  and  durations  to  determine  dates.  DOD 
concurred  with  our  recommendation,  and  we  discuss  later  in  this  report 
the  status  of  DOD’s  efforts  to  address  this  recommendation. 


12An  increment  is  a  useful  and  supportable  operational  capability  that  can  be  developed, 
tested,  produced,  deployed,  and  sustained.  It  may  consist  of  multiple  capability  releases  to 
facilitate  delivery  of  the  system. 

13GAO-1 1-53. 

14A  horizontally  integrated  schedule  links  products  and  outcomes  with  other  associated 
sequenced  activities,  and  a  vertically  integrated  schedule  ensures  that  the  start  and 
completion  dates  for  activities  are  aligned  with  such  dates  on  subsidiary  schedules 
supporting  tasks  and  subtasks. 

15Float  is  the  amount  of  time  by  which  a  predecessor  activity  can  slip  before  the  delay 
affects  the  program’s  estimated  finish  date. 


Page  4 


GAO-14-152  DOD  Business  Systems  Modernization 


GAO’s  Cost  and  Schedule 
Guides 


In  May  2012,  we  issued  the  Schedule  Guide  as  a  companion  to  the  Cost 
Guide.17  A  consistent  methodology  for  developing,  managing,  and 
evaluating  capital  program  cost  estimates  includes  the  concept  of 
scheduling  the  necessary  work  to  a  timeline,  as  discussed  in  the  Cost 
Guide.  Simply  put,  schedule  variances  are  usually  followed  by  cost 
variances.  Because  some  program  costs,  such  as  labor,  supervision, 
rented  equipment,  and  facilities,  cost  more  if  the  program  takes  longer,  a 
reliable  schedule  can  contribute  to  an  understanding  of  the  cost  impact  if 
the  program  does  not  finish  on  time.  In  addition,  management  tends  to 
respond  to  schedule  delays  by  adding  more  resources  or  authorizing 
overtime.  Further,  a  schedule  risk  analysis  allows  for  program 
management  to  account  for  the  cost  effects  of  schedule  slippage  when 
developing  the  life-cycle  cost  estimate.  A  cost  estimate  cannot  be 
considered  fully  credible  if  it  does  not  account  for  the  cost  effects  of 
schedule  slippage. 

A  well-planned  schedule  is  a  fundamental  management  tool  that  can  help 
government  programs  use  public  funds  effectively  by  specifying  when 
work  will  be  performed  in  the  future  and  measuring  program  performance 
against  an  approved  plan.  Moreover,  as  a  model  of  time,  an  integrated 
and  reliable  schedule  can  show  when  major  events  are  expected  to  occur 
as  well  as  the  completion  dates  for  all  activities  leading  up  to  them,  which 
can  help  determine  if  the  program’s  parameters  are  realistic  and 
achievable.  A  program’s  success  depends  in  part  on  the  quality  of  its 
schedule. 


In  March  2009,  we  published  the  Cost  Guide  to  address  a  gap  in  federal 
guidance  about  processes,  procedures,  and  practices  needed  for 
ensuring  reliable  cost  estimates.16  The  Cost  Guide  provides  a  consistent 
methodology  based  on  best  practices  that  can  be  used  across  the  federal 
government  to  develop,  manage,  and  evaluate  capital  program  cost 
estimates.  The  methodology  is  a  compilation  of  characteristics  and 
associated  best  practices  that  federal  cost  estimating  organizations  and 
industry  use  to  develop  and  maintain  reliable  cost  estimates  throughout 
the  life  of  an  acquisition  program. 


16GAO-09-3SP. 

17GAO-12-120G. 
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DEAMS  Program 
Schedule  Did  Not 
Meet  Best  Practices, 
but  the  Cost  Estimate 
Did  Meet  Best 
Practices 


We  found  that  the  schedule  for  the  DEAMS  program  did  not  meet  best 
practices.  The  cost  estimate  did  meet  best  practices,  but  the  issues 
associated  with  the  schedule  could  negatively  affect  the  cost  estimate. 
Specifically,  the  DEAMS  schedule  supporting  the  February  2012 
Milestone  B  decision  partially  or  minimally  met  the  four  characteristics  for 
developing  a  high-quality  and  reliable  schedule — it  was  not 
comprehensive,  well-constructed,  credible,  or  controlled.  In  addition,  our 
assessment  of  the  October  2012  updated  schedule  found  that  it  was  not 
comprehensive,  well-constructed,  and  credible  and  thus  was  also  not 
reliable.18  In  contrast,  the  DEAMS  cost  estimate  fully  or  substantially  met 
the  four  characteristics  of  a  high-quality  and  reliable  cost  estimate — it  was 
comprehensive,  well-documented,  accurate,  and  credible.  However, 
because  the  cost  estimate  is  based  on  the  schedule,  the  unreliability  of 
the  schedule  could  affect  the  cost  estimate.  For  example,  if  there  are 
schedule  slippages,  the  costs  for  the  program  could  be  greater  than 
currently  estimated. 


Program  Schedule  Was 
Not  Developed  in 
Accordance  with  Best 
Practices 


Our  analysis  found  that  the  DEAMS  program  partially  met  three  and 
minimally  met  one  of  the  characteristics  of  a  reliable  schedule  estimate 
and  therefore  did  not  provide  the  information  needed  to  support  the 
February  2012  Milestone  B  decision  (see  table  1).  Appendix  I  contains 
our  detailed  analysis  of  the  DEAMS  schedule  estimate.  The  success  of 
any  program  depends  on  having  a  reliable  schedule  of  the  program’s 
work  activities  that  will  occur,  how  long  they  will  take,  and  how  the 
activities  are  related  to  one  another.  As  such,  the  schedule  not  only 
provides  a  roadmap  for  systematic  execution  of  a  program,  but  also 
provides  the  means  by  which  to  gauge  progress,  identify  and  address 
potential  problems,  and  promote  accountability. 


18We  did  not  assess  the  extent  to  which  the  updated  October  2012  schedule  was 
controlled. 
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Table  1:  Extent  DEAMS  Schedule  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Partially  met 

Well-constructed 

Partially  met 

Credible 

Minimally  met 

Controlled 

Partially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Air  Force. 


Note:  GAO’s  methodology  includes  five  levels  of  compliance  with  its  best  practices.  “Not  met”  means 
the  program  provided  no  evidence  that  satisfies  any  of  the  criterion.  “Minimally  met”  means  the 
program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met”  means  the 
program  provided  evidence  that  satisfies  about  half  of  the  criterion.  “Substantially  met”  means  the 
program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the 
program  provided  evidence  that  completely  satisfies  the  criterion. 


Comprehensive.  A  schedule  should  reflect  all  activities  as  defined  in  the 
program’s  work  breakdown  structure,  including  activities  to  be  performed 
by  the  government  and  the  contractor;  the  resources  (e.g.,  labor, 
materials,  and  overhead)  needed  to  do  the  work;  and  how  long  each 
activity  will  take. 

We  found  that  the  schedule  used  to  support  the  Milestone  B  decision 
included  the  activities  to  be  performed  by  both  the  government  and 
contractor  for  Releases  1  through  3  of  Increment  1.  However,  the 
schedule  did  not  reflect  activities  to  be  performed  for  Releases  4  through 
6  of  Increment  1  and  for  Releases  1  and  2  of  Increment  2.  The  DEAMS 
Program  Manager  stated  that  a  comprehensive  schedule  for  Increment  1 
that  included  the  activities  for  all  six  releases  would  not  be  completed  until 
mid-2014.  The  Program  Manager  also  stated  that  Increment  2  had  not 
been  included  because  program  officials  did  not  know  the  detailed 
activities  to  be  performed  that  far  in  advance.  To  address  this  issue,  the 
DEAMS  program  office  developed  a  roadmap  depicting  Releases  1 
through  6  of  Increment  1  and  Releases  1  and  2  of  Increment  2  with  a  full 
deployment  date  of  fiscal  year  201 7. 19  However,  the  program  office  did 
not  provide  a  schedule  that  supported  the  estimated  dates  in  the 
roadmap. 


1 Q 

A  roadmap  is  a  planning  document  that  briefly  outlines  the  program’s  key  phases 
(increments  and  releases)  and  the  expected  milestones  for  completion.  A  roadmap  does 
not  include  specific  activities  or  a  detailed  work  breakdown  structure,  which  would  be 
included  in  the  schedule  that  is  prepared  to  support  the  roadmap. 
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A  comprehensive  schedule  should  reflect  all  of  a  program’s  activities  and 
recognize  that  uncertainties  and  unknown  factors  in  schedule  estimates 
can  stem  from,  among  other  things,  data  limitations.  As  such,  a  schedule 
incorporates  different  levels  of  detail  depending  on  the  information 
available  at  any  point  in  time.  That  is,  near-term  effort  will  be  planned  in 
greater  detail  than  long-term  effort.  Effort  beyond  the  near  term  that  is 
less  well  defined  is  represented  within  the  schedule  as  long-term  planning 
packages.  Planning  packages  are  a  summarization  of  the  work  to  be 
performed  in  the  distant  future  with  less  specificity.  Planning  packages 
are  planned  at  higher  levels  such  that  a  single  activity  may  represent 
several  months  of  effort,  generic  work  to  be  accomplished,  or  even  a 
future  contract  or  phase.  Planning  packages  can  be  used  as  long  as  they 
are  defined  and  estimated  as  well  as  possible.  By  not  including  all  work 
for  all  deliverables  for  both  increments  and  all  releases,  the  DEAMS 
program  could  incur  difficulties  resulting  from  an  incomplete 
understanding  of  the  plan  and  what  constitutes  a  successful  conclusion 
for  the  program.  DEAMS  program  officials  provided  a  draft  of  the 
Schedule  Management  Plan  that  documented  their  intent  to  use  a 
planning  package  approach  when  updating  the  DEAMS  schedule  in  the 
future. 

Resources  were  identified  in  the  schedule;  however,  the  resources  were 
not  assigned  to  specific  activities  in  the  schedule.  Although  our  analysis 
determined  that  activity  durations  were  manageable  and  reasonably 
estimated,  resource  availability  affects  estimates  of  work  and  its  duration, 
as  well  as  resources  that  will  be  available  for  subsequent  activities. 
DEAMS  program  management  officials  told  us  that  government  resource 
allocations  are  determined  by  management  as  needed.  These  officials 
told  us  that  management  does  not  necessarily  take  into  consideration  the 
resource  information  captured  in  the  schedule  when  determining  resource 
allocations.  However,  DEAMS  officials  did  not  provide  any  documentation 
that  specific  resources  were  being  mapped  to  the  schedule.  As 
mentioned  above,  the  estimates  of  work  required  and  duration  for  an 
activity  are  tied  to  the  availability  of  resources;  therefore,  the  lack  of  such 
information  could  hinder  management’s  ability  to  compute  total  labor  and 
equipment  hours,  calculate  total  project  and  per-period  cost,  resolve 
resource  conflicts,  and  establish  the  reasonableness  of  the  plan. 

Well-constructed.  A  schedule  should  be  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this  objective,  all  activities  should  be  logically 
sequenced — that  is,  listed  in  the  order  in  which  they  are  to  be  carried  out. 
In  particular,  activities  that  must  finish  prior  to  the  start  of  other  activities 
(i.e. ,  predecessor  activities),  as  well  as  activities  that  cannot  begin  until 
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other  activities  are  completed  (i.e.,  successor  activities),  should  be 
identified  and  their  relationships  established.  The  establishment  of  a 
critical  path  is  necessary  for  examining  the  effects  of  any  activity  slipping 
along  this  path.  The  calculation  of  a  critical  path  determines  which 
activities  drive  the  project’s  earliest  completion  date.  The  schedule  should 
also  identify  total  float  so  that  the  schedule’s  flexibility  can  be  accurately 
determined. 

We  found  that  the  majority  of  logic  used  to  sequence  the  activities  within 
the  schedule  was  generally  error-free  with  a  minimal  use  of  lags,20  clearly 
indicating  to  program  management  the  order  of  activities  that  must  be 
accomplished.  Although  we  found  few  missing  logic  relationships  for 
Release  3  of  Increment  I,21  approximately  25  percent  of  the  remaining 
activities  for  Releases  1  and  2  of  Increment  1  were  missing  logic 
relationships.  Because  interdependencies  among  activities  were  not 
identified,  the  DEAMS  program  management  officials’  ability  to  properly 
calculate  dates  and  predict  changes  in  the  future  is  impaired.  We  found  a 
significant  number  of  constraints  for  activities  throughout  the  schedule.22 
A  schedule  is  intended  to  be  a  dynamic,  proactive  planning  and  risk 
mitigation  tool  that  models  the  program  and  can  be  used  to  track  progress 
toward  important  milestones.  Schedules  with  constrained  dates  can 
portray  an  artificial  or  unrealistic  view  of  the  project.  Constraints  should  be 
minimized  because  they  can  create  false  dates  in  a  schedule. 

Further,  the  schedule  did  not  have  a  valid  critical  path  and  identified 
critical  activities  more  by  their  constraints  than  by  logic.  Rather  than 
relying  on  constraints,  the  schedule  should  use  logic  and  durations  in 
order  to  reflect  realistic  start  and  completion  dates  for  activities. 
Successfully  identifying  the  critical  path  relies  on  several  factors,  such  as 
capturing  all  activities,  properly  sequencing  activities,  and  assigning 
resources,  which,  as  noted  earlier,  had  not  been  done.  Without  a  valid 


on 

A  lag  in  a  schedule  denotes  the  passage  of  time  between  two  activities.  Lags  have  a 
specific  use  in  scheduling  but  may  be  misused  to  force  activities  to  begin  on  specific 
dates. 

21The  purpose  of  a  logic  relationship,  or  dependency,  is  to  depict  the  sequence  in  which 
activities  occur.  Such  relationships  state  when  activities  are  planned  to  start  and  finish  in 
relation  to  the  start  and  finish  of  other  activities.  A  logic  relationship  therefore  models  the 
effect  of  an  on-time,  delayed,  or  accelerated  activity  on  subsequent  activities. 

22 A  constraint  predefines  the  start,  finish,  or  both  dates  of  an  activity. 
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critical  path,  management  cannot  focus  on  activities  that  will  have 
detrimental  effects  on  the  key  project  milestones  and  deliveries  if  they 
slip. 

We  found  that  total  float  was  not  reasonable,  and  that  in  some  instances 
unreasonable  float  was  a  direct  result  of  improper  sequencing  or  missing 
logic.  Releases  1  and  2  of  Increment  1  showed  that  25  percent  of 
program  activities  had  total  float  equal  to  or  greater  than  392  working 
days,  meaning  that  those  activities  could  slip  almost  2  working  years  and 
not  affect  the  end  date  of  the  program.  Without  knowledge  of  the  reason 
float  exists  for  a  program  activity,  management  cannot  determine  the 
flexibility  of  tasks  and  therefore  cannot  properly  reallocate  resources  from 
tasks  that  can  safely  slip  to  tasks  that  cannot  slip  without  adversely 
affecting  the  estimated  program  completion  date. 

Credible.  A  schedule  should  be  horizontally  and  vertically  integrated.  A 
horizontally  integrated  schedule  links  products  and  outcomes  with  other 
associated  sequenced  activities,  which  helps  verify  that  activities  are 
arranged  in  the  right  order  to  achieve  aggregated  products  or  outcomes. 
A  vertically  integrated  schedule  ensures  that  the  start  and  completion 
dates  for  activities  are  aligned  with  such  dates  on  subsidiary  schedules 
supporting  tasks  and  subtasks.  Such  mapping  or  alignment  among 
subsidiary  schedules  enables  different  groups — such  as  government 
teams  and  contractors — to  work  to  the  same  master  schedule,  and 
provides  assurance  that  the  representation  of  the  schedule  to  different 
audiences  is  consistent  and  accurate.  A  schedule  risk  analysis  should 
also  be  performed  using  statistical  techniques  to  predict  the  level  of 
confidence  in  meeting  a  program’s  completion  date. 

We  found  that  Release  3  of  Increment  1  exhibited  horizontal  integration, 
but  Releases  1  and  2  of  Increment  1  did  not  because  date  constraints 
prevented  forecasted  dates  from  being  calculated  realistically  for  future 
activities.  If  the  schedule  lacks  horizontal  integration,  activities  whose 
durations  are  greatly  extended  will  have  no  effect  on  key  milestones 
reflected  in  the  schedule.  We  further  found  that  Releases  1  and  2  of 
Increment  1  did  not  demonstrate  vertical  integration.  For  example,  we 
found  instances  where  the  start  dates  for  the  same  activities  differed  by  1 
day,  1  week,  and  1  month  between  the  government  and  contractor 
schedules.  Unless  the  schedule  is  vertically  integrated,  lower-level 
schedules  will  not  be  consistent  with  upper-level  schedule  milestones, 
affecting  the  integrity  of  the  entire  schedule  and  the  ability  of  different 
teams  to  work  to  the  same  schedule  expectations. 


Page  10 


GAO-14-152  DOD  Business  Systems  Modernization 


DEAMS  program  management  officials  stated  that  a  schedule  risk 
analysis  had  not  been  conducted  because  the  schedule  had  not  been 
approved  to  be  used  as  a  baseline  schedule — the  target  schedule  against 
which  program  performance  can  be  measured,  monitored,  and  reported. 
These  officials  stated  that  although  this  analysis  had  not  been  conducted, 
they  were  collecting  best-case  and  worst-case  durations  from  the 
contractor  with  their  periodic  schedule  delivery.  These  data  can  be  used 
by  program  management  to  calculate  more  reliable  estimates  of  durations 
for  future  activities.  However,  we  found  that  the  schedule  did  not  contain 
best-  or  worst-case  duration  data  for  600  of  605  detailed  activities.  For  the 
five  instances  where  duration  data  were  contained  in  the  schedule,  we 
determined  that  four  were  questionable  because  two  activities  were 
already  completed  and  two  had  already  exceeded  the  worst-case 
estimate.  If  a  schedule  risk  analysis  is  not  conducted,  program 
management  cannot  determine  the  likelihood  of  the  project’s  completion 
date,  how  much  schedule  risk  contingency  is  needed  to  provide  an 
acceptable  level  of  certainty  for  completion  by  a  specific  date,  risks  most 
likely  to  delay  the  project,  how  much  contingency  reserve  each  risk 
requires,  and  the  paths  or  activities  that  are  most  likely  to  delay  the 
project.  As  discussed  later,  the  lack  of  a  schedule  risk  analysis  can  affect 
the  credibility  of  the  cost  estimate. 

Controlled.  A  schedule  should  be  continuously  updated  using  logic, 
durations,  and  actual  progress  to  realistically  forecast  dates  for  program 
activities.  A  schedule  narrative  should  accompany  the  updated  schedule 
to  provide  decision  makers  and  auditors  a  log  of  changes  and  their  effect, 
if  any,  on  the  schedule  time  frame.  The  schedule  should  be  analyzed 
continuously  for  variances  to  determine  when  forecasted  completion 
dates  differ  from  planned  dates.  This  analysis  is  especially  important  for 
those  variations  that  affect  activities  identified  as  being  in  a  program’s 
critical  path  and  that  can  affect  a  scheduled  completion  date.  A  baseline 
schedule  should  be  used  to  manage  the  program  scope,  the  time  period 
for  accomplishing  it,  and  the  required  resources. 

We  found  that  DEAMS  program  management  met  weekly  to  discuss 
proposed  schedule  changes  and  updated  the  schedule’s  progress. 
However,  a  schedule  narrative  was  not  prepared  by  DEAMS  program 
management.  In  addition,  we  found  a  number  of  date  anomalies 
throughout  the  schedule,  including  activities  with  planned  start  dates 
scheduled  to  occur  in  the  past  and  activities  with  actual  finish  dates 
scheduled  to  occur  in  the  future.  We  also  found  a  number  of  out-of- 
sequence  activities  in  the  schedule — activities  that  started  before  their 
predecessors  finished,  in  contradiction  to  the  planned  sequence.  If  the 
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schedule  is  not  continually  monitored  to  determine  when  forecasted 
completion  dates  differ  from  planned  dates,  then  it  cannot  be  used  to 
determine  whether  schedule  variances  will  affect  work  needed  to  be 
accomplished  at  a  future  date. 

We  also  found  that  there  was  no  baseline  schedule  that  could  be  used  to 
measure  program  performance.  DEAMS  program  management  officials 
did  maintain  a  schedule  narrative  document  that  contained  a  list  of 
custom  fields  and  assumptions;  however,  the  document  did  not  explain 
ground  rules  and  assumptions,  justifications  for  logic,  and  other  unique 
features  of  the  schedule.  These  officials  stated  that  other  process 
documents  were  being  developed.  Without  a  formally  established 
baseline  schedule  to  measure  performance  against,  management  cannot 
identify  or  mitigate  the  effect  of  unfavorable  performance. 


Updated  Schedule  Also  Our  assessment  of  the  updated  schedule  dated  October  2012  found  that 

Did  Not  Follow  Best  it  was  not  comprehensive,  well-constructed  and  credible.23  Although  the 

Practices  DEAMS  Program  Manager  stated  that  the  government  and  contractor 

activities  for  Releases  1  through  3  of  Increment  1  had  been  integrated  in 
the  October  2012  schedule,  this  schedule  was  not  comprehensive. 
Specifically,  it  excluded  activities  for  both  the  government  and  contractor 
related  to  Releases  4  through  6  of  Increment  1  and  Releases  1  and  2  of 
Increment  2.  If  activities  are  missing  from  the  schedule,  then  other  best 
practices  will  not  be  met.  The  schedule  was  also  missing  relationships  for 
a  significant  number  of  the  remaining  milestones  and  activities.  In 
addition,  the  October  2012  schedule  included  a  significant  number  of  date 
constraints  with  little  or  no  justification  for  their  use  in  the  schedule. 

Similar  to  the  previous  schedule,  the  updated  schedule  presented 
unreasonable  float  throughout  and  did  not  include  a  schedule  risk 
analysis.  As  a  result  of  these  shortcomings,  the  updated  schedule  was 
not  reliable.  Further,  program  officials  could  not  rely  on  this  schedule  as  a 
baseline  to  effectively  manage  and  monitor  program  performance. 

In  May  2013,  program  management  officials  provided  another  updated 
DEAMS  schedule  that  they  stated  included  some  improvements,  but  they 
acknowledged  that  it  contained  issues  that  prevented  the  schedule  from 


oo 

Program  officials  stated  that  the  updated  schedule  represented  the  DEAMS  program 
baseline  as  of  October  2012.  The  updated  schedule  included  the  period  May  2,  2011,  to 
December  17,  2015,  and  encompassed  Releases  1  through  3  of  Increment  1. 
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meeting  best  practices.  For  example,  these  officials  stated  that  the  May 
2013  schedule  included  long-term  planning  packages  for  activities  related 
to  Releases  4  through  6  of  Increment  1  and  Releases  1  and  2  of 
Increment  2,  integrated  government  and  contractor  activities,  and 
reduced  the  number  of  constraints  and  out-of-sequence  activities  in  the 
schedule.  However,  the  officials  acknowledged  that  several  outstanding 
issues  remained  related  to,  for  example,  vertical  and  horizontal 
integration,  missing  logic  relationships,  and  the  lack  of  a  schedule  risk 
analysis.  Although  we  did  not  independently  assess  the  May  2013 
schedule  to  determine  whether  it  met  the  four  schedule  characteristics, 
we  did  confirm  that  it  included  long-term  planning  packages,  which  are 
needed  to  create  a  complete  picture  of  the  program  from  start  to  finish 
and  to  allow  the  monitoring  of  a  program’s  critical  path.  The  results  of  our 
analyses  of  the  schedule  that  supported  the  February  2012  Milestone  B 
decision  and  October  2012  DEAMS  schedule  reflect  similar  weaknesses 
to  those  we  reported  in  October  2010. 24  Therefore,  given  the  findings  of 
this  review,  our  prior  recommendation  for  improving  the  DEAMS  schedule 
remains  valid. 


Program  Cost  Estimate  We  found  that  the  DEAMS  program  fully  or  substantially  met  the  four 

Met  Best  Practices  characteristics  of  a  reliable  cost  estimate  to  support  the  Milestone  B 

decision,  as  shown  in  table  2.  However,  because  the  cost  estimate  relies 
on  dates  derived  from  the  schedule  and  we  are  questioning  the  reliability 
of  the  forecasted  program  dates,  the  credibility  of  the  cost  estimate  can 
be  affected.  Appendix  II  contains  our  detailed  analysis  of  the  DEAMS  cost 
estimate.  A  reliable  cost  estimate  is  critical  to  the  success  of  any 
program.  Such  an  estimate  provides  the  basis  for  informed  investment 
decision  making,  realistic  budget  formulation  and  program  resourcing, 
meaningful  progress  measurement,  proactive  course  correction  when 
warranted,  and  accountability  for  results. 


24GAO-1 1-53. 
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Table  2:  Extent  DEAMS  Cost  Estimate  Met  Best  Practices 

Characteristic 

Assessment 

Comprehensive 

Fully  met 

Well-documented 

Substantially  met 

Accurate 

Substantially  met 

Credible 

Substantially  met 

Source:  GAO  analysis  based  on  information  provided  by  the  Air  Force. 


Note:  GAO’s  methodology  includes  five  levels  of  compliance  with  its  best  practices.  “Not  met”  means 
the  program  provided  no  evidence  that  satisfies  any  of  the  criterion.  “Minimally  met”  means  the 
program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met”  means  the 
program  provided  evidence  that  satisfies  about  half  of  the  criterion.  “Substantially  met”  means  the 
program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the 
program  provided  evidence  that  completely  satisfies  the  criterion. 


Comprehensive.  A  cost  estimate  should  include  costs  of  the  program  over 
its  full  life  cycle,  provide  a  level  of  detail  appropriate  to  ensure  that  cost 
elements  are  neither  omitted  nor  double-counted,  and  document  all  cost- 
influencing  ground  rules  and  assumptions.  The  cost  estimate  should  also 
completely  define  the  program  and  be  technically  reasonable. 

We  found  that  the  cost  estimate  for  DEAMS  was  comprehensive.  The 
cost  estimate  included  both  government  and  contractor  costs  of  the 
program  over  its  life  cycle — from  the  inception  of  the  program  through 
design,  development,  deployment,  and  operation  and  maintenance — as 
outlined  in  the  roadmap  prepared  by  program  officials.  As  stated  earlier, 
the  roadmap  provided  an  overall  summary  of  the  program’s  key  phases 
(increments  and  releases)  and  the  expected  milestones  for  completion. 
The  cost  estimate  also  included  an  appropriate  level  of  detail,  which 
provided  assurance  that  cost  elements  were  neither  omitted  nor 
duplicated,  and  included  documentation  of  all  cost-influencing  ground 
rules  and  assumptions.  The  cost  estimate  documentation  included  the 
purpose  of  the  cost  estimate,  a  technical  description  of  the  program,  and 
technical  risks  (e.g.,  the  resolution  for  any  identified  deficiencies). 

Well-documented.  A  cost  estimate  should  be  supported  by  detailed 
documentation  that  describes  how  it  was  derived  and  how  the  expected 
funding  will  be  spent  in  order  to  achieve  a  given  objective.  Therefore,  the 
documentation  should  capture  such  things  as  the  source  data  used,  the 
calculations  performed,  the  results  of  the  calculations,  the  estimating 
methodology  used  to  derive  each  work  breakdown  structure  element’s 
cost,  and  evidence  that  the  estimate  was  approved  by  management. 
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We  found  that  the  cost  estimate  for  DEAMS  was  well-documented.  The 
cost  estimate  captured  such  things  as  the  source  data  used,  the 
calculations  performed  and  the  results  of  the  calculations,  and  the 
rationale  for  choosing  a  particular  estimating  methodology.  This 
information  was  captured  in  such  a  way  that  the  data  used  to  derive  the 
estimate  can  be  traced  back  to,  and  verified  against,  the  sources  so  that 
the  estimate  can  be  easily  replicated.  However,  there  was  no  discussion 
of  efforts  taken,  if  any,  to  ensure  the  reliability  of  the  data  used.  The 
DEAMS  Program  Management  Office  presented  evidence  of  receiving 
approval  of  the  estimate  through  briefings  to  management. 

Accurate.  A  cost  estimate  should  be  based  on  an  assessment  of  most 
likely  costs  (adjusted  properly  for  inflation),  updated  to  reflect  significant 
changes  and  grounded  in  a  historical  record  of  cost  estimating  and  actual 
experiences  on  other  comparable  programs. 

We  found  that  the  cost  estimate  for  DEAMS  was  accurate.  The  cost 
estimate  provided  results  that  were  substantially  unbiased,  and  the  cost 
model  detailed  the  calculations  and  inflation  indexes  underlying  the 
estimate.  Calculations  within  the  model  could  be  traced  back  to 
supporting  documentation.  The  cost  estimate  was  updated  regularly  to 
reflect  significant  changes  in  the  program  and  updated  annually  to 
incorporate  actual  costs  expended  in  prior  fiscal  years.  Further,  the  cost 
estimate  was  based  on  historical  data.  However,  the  cost  estimate  did  not 
discuss  variances  between  planned  and  actual  costs,  which  would  enable 
estimators  to  assess  how  well  they  are  estimating  program  costs  and  to 
identify  lessons  learned. 

Credible.  A  cost  estimate  should  discuss  any  limitations  of  the  analysis 
because  of  uncertainty  or  biases  surrounding  data  or  assumptions.  In 
addition,  the  estimate’s  results  should  be  cross-checked  and  reconciled  to 
an  independent  cost  estimate  to  determine  whether  other  estimating 
methods  produce  similar  results. 

We  found  that  the  cost  estimate  was  credible.  The  DEAMS  Program 
Management  Office  conducted  a  risk  and  uncertainty  analysis  by 
identifying  the  cost  elements  with  the  greatest  degree  of  uncertainty, 
determining  the  cost  drivers  for  the  program,  and  identifying  the  impact  of 
changing  major  ground  rules  and  cost  driver  assumptions.  An 
independent  cost  estimate  developed  by  the  Air  Force  Cost  Analysis 
Agency  was  reconciled  to  the  program’s  estimate.  However,  a  sensitivity 
analysis  was  not  completed  for  each  of  the  major  cost  drivers.  As  a  result, 
the  cost  estimator  will  not  have  a  clear  understanding  of  how  each  major 
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cost  driver  is  affected  by  a  change  in  a  single  assumption  and  thus  which 
scenario  most  affects  the  cost  estimate.  Further,  as  discussed  previously, 
because  a  schedule  risk  analysis  was  not  performed  as  required  by  best 
practices,  the  cost  estimate  does  not  include  a  contingency  amount  to 
account  for  any  schedule  slippage  that  could  occur.  To  the  extent  that  a 
schedule  slippage  does  occur,  there  could  ultimately  be  an  impact  on  the 
cost  estimate. 

Conclusions 

i 

The  Air  Force  did  not  meet  best  practices  in  developing  a  schedule  for  the 
DEAMS  program.  As  a  result,  this  raises  questions  about  the  credibility  of 
the  deadline  for  acquiring  and  implementing  DEAMS  to  provide  needed 
functionality  for  financial  improvement  and  audit  readiness.  Because  of 
these  questions,  the  cost  estimate,  while  following  best  practices,  may  not 
fully  capture  all  costs  associated  with  the  program,  particularly  if  there  is 
significant  schedule  slippage.  Moreover,  Air  Force  management  did  not 
have  a  reliable  schedule  estimate  when  making  its  decision  to  invest  in 
the  DEAMS  program.  It  is  critical  to  correct  the  deficiencies  identified  with 
the  schedule  estimate  to  help  ensure  that  the  projected  spending  for  this 
program  is  being  used  in  the  most  efficient  and  effective  manner. 

Recommendation  for 
Executive  Action 

i 

To  help  provide  for  the  successful  implementation  of  DEAMS,  we 
recommend  that  the  Secretary  of  the  Air  Force  direct  the  Under  Secretary 
of  the  Air  Force,  in  his  capacity  as  the  Chief  Management  Officer,  to 
consider  and  make  any  necessary  adjustments  to  the  DEAMS  cost 
estimate  after  addressing  our  prior  recommendation  to  adopt  scheduling 
best  practices. 

Agency  Comments 

i 

We  provided  a  draft  of  this  report  to  DOD  for  review  and  comment.  In  its 
written  comments,  reprinted  in  appendix  III,  DOD  concurred  with  our 
recommendation.  DOD  also  provided  a  technical  comment,  which  we 
incorporated. 

As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  to  the  appropriate 
congressional  committees,  the  Secretary  of  Defense;  the  Secretary  of  the 
Air  Force;  the  Assistant  Secretary  of  Defense  (Acquisition);  the  Deputy 
Chief  Management  Officer;  the  Under  Secretary  of  Defense  (Comptroller); 
the  Under  Secretary  of  the  Air  Force,  in  his  capacity  as  the  Chief 
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Management  Officer  of  the  Air  Force;  and  the  Program  Manager  for 
DEAMS.  In  addition,  the  report  will  be  available  at  no  charge  on  the  GAO 
website  at  http://www.gao.gov. 

If  you  or  your  staffs  have  any  questions  about  this  report,  please  contact 
Asif  A.  Khan  at  (202)  512-9869  or  khana@gao.gov  or  Nabajyoti  Barkakati 
at  (202)  512-4499  or  barkakatin@gao.gov.  Contact  points  for  our  Offices 
of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last 
page  of  this  report.  GAO  staff  members  who  made  key  contributions  to 
this  report  are  listed  in  appendix  IV. 


Asif  A.  Khan 
Director 

Financial  Management  and  Assurance 


Nabajyoti  Barkakati 

Chief  Technologist 

Applied  Research  and  Methods 

Center  for  Science,  Technology,  and  Engineering 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 
schedule  estimate  supporting  the  February  2012  Milestone  B  decision1 
met  the  characteristics  of  a  high-quality,  reliable  schedule.  Table  3 
provides  the  detailed  results  of  our  analysis. 

GAO’s  methodology  includes  five  levels  of  compliance  with  its  best 
practices.2  “Not  met”  means  the  program  provided  no  evidence  that 
satisfies  any  of  the  criterion.  “Minimally  met”  means  the  program  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met” 
means  the  program  provided  evidence  that  satisfies  about  half  of  the 
criterion.  “Substantially  met”  means  the  program  provided  evidence  that 
satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the  program 
provided  evidence  that  completely  satisfies  the  criterion. 


Milestone  B  is  considered  the  official  start  of  the  program  according  to  the  Defense 
Acquisition  Management  System  Framework.  The  framework  is  intended  to  translate 
mission  needs  and  requirements  into  stable,  affordable,  and  well-managed  acquisition 
programs. 

2GAO,  GAO  Schedule  Assessment  Guide:  Best  Practices  for  Project  Schedules — 
Exposure  Draft,  GAO-12-120G  (Washington,  D.C.:  May  2012). 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


Table  3:  Assessment  of  DEAMS  Schedule  Estimate  Compared  to  Best  Practices 


Characteristic  Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Comprehensive  The  schedule  should  reflect  all  Partially  met 
activities  as  defined  in  the 
project’s  work  breakdown 
structure,  which  defines  in  detail 
the  work  necessary  to 
accomplish  a  project’s  objectives, 
including  activities  to  be 
performed  by  both  the 
government  and  contractors.  The 
schedule  should  reflect  the 
resources  (e.g.,  labor,  materials, 
and  overhead)  needed  to  do  the 
work,  whether  those  resources 
will  be  available  when  needed, 
and  whether  any  funding  or  time 
constraints  exist.  The  schedule 
should  realistically  reflect  how 
long  each  activity  will  take, 
allowing  for  discrete  progress 
measurement  with  specific  start 
and  finish  dates. 


The  schedule  reflected  the  activities  to  be  performed  by  both 
the  government  and  the  contractor  for  Increment  1,  Releases 
1  through  3,  of  the  program.  However,  it  did  not  reflect 
activities  for  Releases  4  through  6  of  Increment  1  and  for 
Releases  1  and  2  of  Increment  2  of  the  program.  To  be 
comprehensive,  the  program  schedule  should  reflect  all  effort 
necessary  to  successfully  complete  the  program.  In  addition, 
the  program  schedule  was  not  fully  integrated.  Specifically, 
although  the  schedule  narrative  noted  that  Release  2  of 
Increment  1  included  contractor  activities  and  that  integration 
with  government  activities  was  in  process,  there  were  no  links 
between  the  government  and  contractor  activities  for  this 
release.  For  the  most  part,  resources  had  been  identified; 
however,  the  resources  had  not  been  assigned  to  specific 
activities  in  the  schedule.  The  amount  of  available  resources 
affects  estimates  of  work  and  its  duration  and  resources  that 
will  be  available  for  subsequent  activities.  Information  on 
resource  needs  and  availability  assists  the  program  office  in 
forecasting  the  likelihood  that  activities  will  be  completed  as 
scheduled.  Because  the  current  schedule  did  not  allow  insight 
into  current  or  projected  assignment  of  resources,  the  risk  of 
program  slippage  was  significantly  increased. 

The  project  duration  was  determined  by  a  target  date  and  the 
majority  of  activity  durations  in  the  schedule  met  the  standard 
best  practice  for  activity  duration  (44  days).  Specifically,  98 
percent  of  the  activities  were  less  than  44  days. 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


Characteristic  Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Well-constructed  The  schedule  should  be  planned  Partially  met 
so  that  critical  project  dates  can 
be  met.  All  activities  should  be 
logically  sequenced — that  is, 
listed  in  the  order  in  which  they 
are  to  be  carried  out — with 
predecessor  and  successor  logic. 

Date  constraints  and  lags  should 
be  minimized  and  justified  to  help 
ensure  that  the  interdependence 
of  activities  that  collectively  lead 
to  the  completion  of  events  or 
milestones  can  be  established 
and  used  to  guide  work  and 
measure  progress.  The  schedule 
should  identify  the  program 
critical  path — the  path  of  longest 
duration  through  the  sequence  of 
activities — to  determine  the 
program’s  earliest  completion 
date.  The  schedule  should 
identify  total  float — the  amount  of 
time  by  which  a  predecessor 
activity  can  slip  before  the  delay 
affects  the  program’s  estimated 
finish  date — so  that  the 
schedule’s  flexibility  can  be 
accurately  determined.  As  a 
general  rule,  activities  along  the 
critical  path  have  the  least 
amount  of  float. 


The  schedule  was  not  planned  to  ensure  that  critical  project 
dates  could  be  met.  The  majority  of  logic  used  to  sequence 
the  activities  within  the  schedule  was  generally  error-free  with 
a  minimal  use  of  lags — the  passage  of  time  between  two 
activities — and  leads — negative  lags  used  to  accelerate  a 
successor  activity,  clearly  indicating  which  activities  must 
finish.  Our  analysis  found  that  the  schedule  for  Increment  1 , 
Release  3,  accounted  for  predecessor  and  successor  logic  on 
97  percent  of  remaining  activities  and  milestones.  However, 
our  analysis  of  the  schedule  for  Releases  1  and  2  of 
Increment  1  showed  that  for  the  remaining  activities  and 
milestones  10  percent  were  missing  predecessor  links,  17 
percent  were  missing  successor  links,  and  2  percent  were 
missing  both.  That  is,  approximately  one-fourth  of  all 
remaining  detail  activities  and  milestones  were  missing  some 
logic  link.  Missing  predecessors  or  successors  reduce  the 
credibility  of  the  calculated  dates.  If  an  activity  that  has  no 
logical  successor  slips,  the  schedule  will  not  reflect  the  effect 
on  the  critical  path,  float,  or  scheduled  start  dates  of 
downstream  activities. 

In  addition,  there  were  a  small  percentage  of  activities  with 
dangling  relationships  in  both  schedules.  A  dangling 
relationship  is  a  scheduling  relationship  that  is  not  properly 
tied  to  an  activity’s  start  or  end  date.  Each  activity’s  start 
date — other  than  the  start  and  finish  milestones — must  be 
driven  by  a  predecessor  activity,  and  each  activity’s  finish  date 
must  drive  a  successor  activity’s  start  or  finish.  The  schedule 
for  Releases  1  and  2  of  Increment  1  had  5  remaining  activities 
whose  start  dates  were  not  being  driven  by  a  predecessor 
activity  and  1 1  activities  that  had  finish  dates  that  did  not  drive 
successor  activities.  The  schedule  for  Release  3  of  Increment 
1  had  1  remaining  activity  whose  start  date  was  not  being 
driven  by  a  predecessor  activity  and  24  activities  that  had 
finish  dates  that  did  not  drive  successor  activities.  Dangling 
relationships,  a  form  of  incomplete  relationships,  can  interfere 
with  the  valid  forecasting  of  scheduled  activities.  Both 
schedules  had  date  constraints  on  about  one-third  of  the 
remaining  activities.  A  customer-mandated  date,  including 
contractual  obligations,  does  not  constitute  a  legitimate 
reason  to  constrain  an  activity.  A  schedule  is  intended  to  be  a 
dynamic,  proactive  planning  and  risk  mitigation  tool  that 
models  the  program  and  can  be  used  to  track  progress  toward 
important  milestones.  Schedules  with  constrained  dates  can 
portray  an  artificial  or  unrealistic  view  of  the  project. 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


Characteristic  Best  practice  Assessment  Key  examples  of  rationale  for  assessment 

Our  analysis  found  that  both  schedules  did  not  have  a  valid 
critical  path.  For  example,  there  were  numerous  critical 
activities  scheduled  to  start  before  the  project  status  date  had 
started.  Successfully  identifying  the  critical  path  relies  on 
capturing  all  activities,  properly  sequencing  activities, 
horizontal  traceability,  the  reasonableness  of  float,  accurate 
status  updates,  and — if  there  are  resource  limitations — 
assigning  resources.  Until  the  schedule  can  produce  a  true 
critical  path,  the  program  office  will  not  be  able  to  provide 
reliable  timeline  estimates  or  identify  when  problems  or 
changes  may  occur  and  their  effect  on  downstream  work. 

Also,  without  a  valid  critical  path,  management  cannot  focus 
on  activities  that  will  have  detrimental  effects  on  the  key 
project  milestones  and  deliveries  if  they  slip. 

Our  analysis  found  that  float  calculations  within  the  schedules 
were  not  reliable  and  that  in  some  instances  unreasonable 
float  was  a  direct  result  of  improper  sequencing  or  missing 
logic.  The  schedule  for  Releases  1  and  2  of  Increment  1  had 
average  total  float  for  the  remaining  detail  and  milestone 
activities  of  239  business  days.  Twenty-five  percent  of 
activities  had  total  float  equal  to  or  greater  than  392  working 
days,  meaning  they  could  slip  almost  2  working  years  and  not 
affect  the  end  date  of  the  project.  In  addition,  77  activities  had 
negative  float,  ranging  from  -1  to  -18  days.  Negative  float 
stems  from  constraining  one  or  more  activities  or  milestones 
in  the  network.  Negative  float  indicates  critical  path  effort  that 
may  require  management  action  such  as  overtime  or 
resequencing  of  work.  The  constraint  should  be  examined  and 
justified,  and  the  resulting  negative  float  should  be  evaluated 
for  reasonableness.  The  schedule  for  Release  3  of  Increment 
1  also  had  an  average  total  float  for  the  remaining  detail  and 
milestone  activities  of  289  business  days;  25  percent  of 
remaining  activities  had  total  float  equal  to  or  greater  than  489 
days.  There  were  no  activities  with  negative  float.  Without 
reasonable  float  estimates,  management  may  be  unable  to 
identify  activities  that  could  be  permitted  to  slip  and  thus 
release  and  reallocate  resources  to  activities  that  require 
more  resources  to  be  completed  on  time. 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


Characteristic  Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Credible  The  schedule  should  be  Minimally  met 

horizontally  integrated,  meaning 
that  it  should  link  products  and 
outcomes  associated  with  other 
sequenced  activities.  These  links 
are  commonly  referred  to  as 
“handoffs”  and  serve  to  verify  that 
activities  are  arranged  in  the  right 
order  to  achieve  aggregated 
products  or  outcomes.  The 
schedule  should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and  completing 
activities  in  the  integrated  master 
schedule  should  be  aligned  with 
the  dates  for  supporting  tasks 
and  subtasks.  Such  mapping  or 
alignment  among  levels  enables 
different  groups  to  work  to  the 
same  master  schedule. 

A  schedule  risk  analysis  should 
be  performed  using  statistical 
techniques  to  predict  the  level  of 
confidence  in  meeting  a 
program’s  completion  date, 
determine  the  time  contingency 
needed  fora  level  of  confidence, 
and  identify  high-priority  risks  and 
opportunities.  The  analysis 
focuses  not  only  on  critical  path 
activities  but  also  on  activities 
near  the  critical  path,  since  they 
can  affect  the  program’s  status. 


The  schedule  for  Release  3  of  Increment  1  exhibited 
horizontal  traceability;  however,  the  schedule  for  Releases  1 
and  2  of  Increment  1  did  not  because  date  constraints 
prevented  forecasted  dates  from  being  calculated  realistically 
for  future  activities.  Unless  the  schedule  is  horizontally 
traceable,  activities  whose  durations  are  greatly  extended  will 
have  no  effect  on  key  milestones. 

Our  analysis  did  not  find  vertical  traceability  within  the 
schedule  for  Releases  1  and  2  of  Increment  1 — the  ability  to 
consistently  trace  work  breakdown  structure  activities 
between  detailed,  intermediate,  and  master  schedules.  For 
example,  we  traced  three  activities  between  the  government 
schedule  and  the  underlying  prime  contractor  schedule  and  in 
each  case  found  mismatching  start  dates  that  differed  by  a 
day,  a  week,  and  a  month.  Vertical  traceability  provides 
assurance  that  the  representation  of  the  schedule  to  different 
audiences  is  consistent  and  accurate.  Unless  the  schedule  is 
vertically  traceable,  lower-level  schedules  will  not  be 
consistent  with  upper-level  schedule  milestones,  affecting  the 
integrity  of  the  entire  schedule  and  the  ability  of  different 
teams  to  work  to  the  same  schedule  expectations. 

The  program  office  had  not  conducted  a  schedule  risk 
analysis.  If  a  schedule  risk  analysis  is  not  conducted,  program 
management  cannot  determine  the  likelihood  of  meeting  the 
program’s  completion  date,  how  much  schedule  risk 
contingency  is  needed  to  provide  an  acceptable  level  of 
certainty  for  completion  by  a  specific  date,  risks  most  likely  to 
delay  the  program,  how  much  contingency  reserve  each  risk 
requires,  and  the  paths  or  activities  that  are  most  likely  to 
delay  the  program.  Unless  a  statistical  simulation  is  run, 
calculating  the  completion  date  from  schedule  logic  and  the 
most  likely  duration  distributions  will  tend  to  underestimate  the 
program’s  overall  critical  path  duration. 
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Appendix  I:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Schedule  Estimate 


Characteristic  Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Controlled  The  schedule  should  be  Partially  met 

continuously  updated  using  logic 
and  durations  to  realistically 
forecast  dates  for  program 
activities.  A  schedule  narrative 
should  accompany  the  updated 
schedule  to  provide  decision 
makers  and  auditors  a  log  of 
changes  and  their  effect,  if  any, 
on  the  schedule  time  frame. 

The  schedule  should  be  analyzed 
continuously  for  variances  to 
determine  when  forecasted 
completion  dates  differ  from 
planned  dates.  This  analysis  is 
especially  important  for  those 
variations  that  impact  activities 
identified  as  being  on  a 
program’s  critical  path  and  can 
impact  a  scheduled  completion 
date.  A  baseline  schedule  is  the 
basis  for  managing  the  program 
scope,  the  time  period  for 
accomplishing  it,  and  the 
required  resources.  The  baseline 
schedule  is  designated  the  target 
schedule,  and  is  subject  to  a 
configuration  management 
control  process,  against  which 
program  performance  can  be 
measured,  monitored,  and 
reported.  A  corresponding 
baseline  document  explains  the 
overall  approach  to  the  program; 
defines  custom  fields  in  the 
schedule  file;  details  ground  rules 
and  assumptions  used  in 
developing  the  schedule;  and 
justifies  constraints,  lags,  long 
activity  durations,  and  any  other 
unique  features  of  the  schedule. 


Management  met  weekly  to  discuss  proposed  schedule 
changes  and  update  schedule  progress.  The  program 
management  office  followed  general  guidelines  for  updating 
the  schedule;  however,  a  schedule  narrative  did  not 
accompany  the  updates,  and  we  found  a  significant  number  of 
date  anomalies  in  the  schedules  provided  for  Releases  1 
through  3  of  Increment  1 .  For  example,  we  found  instances  of 
activities  with  start  dates  in  the  past,  activities  with  planned 
finish  dates  in  the  past,  activities  with  actual  start  dates  in  the 
future,  and  activities  with  actual  finish  dates  in  the  future.  We 
also  found  a  number  of  out-of-sequence  activities,  that  is, 
activities  that  started  before  their  predecessors  finished.  A 
schedule  is  a  fundamental  program  management  tool  that 
specifies  when  work  will  be  performed  in  the  future  and  how 
well  the  program  is  performing  against  an  approved  plan.  If 
the  schedule  is  not  updated,  then  it  is  impossible  to  tell  what 
activities  have  been  completed,  are  in  progress,  are  late,  and 
are  planned  to  start  on  time.  In  addition,  if  a  schedule  is  not 
updated  to  reflect  what  is  actually  occurring  on  the  program,  it 
may  have  inaccurate  completion  dates  and  critical  paths. 
When  this  is  the  case,  management  cannot  use  the  schedule 
to  monitor  progress  and  make  decisions  regarding  risk 
mitigation  and  resource  allocations. 

There  was  no  baseline  schedule  that  could  be  used  to 
measure  program  performance.  As  a  result,  it  was  not 
possible  to  track  variances  to  past  baselines  using  the 
schedules  provided  for  Increment  1,  Releases  1  through  3. 

The  program  management  office  did  maintain  a  baseline 
schedule  narrative  document  that  contained  a  list  of  custom 
fields  and  assumptions.  However,  this  document  did  not 
explain  the  overall  program  approach,  ground  rules  and 
assumptions,  justifications  for  logic,  and  other  unique  features 
of  the  schedule.  Without  a  formally  established  baseline 
schedule  to  measure  performance  against,  management 
cannot  identify  or  mitigate  the  effect  of  unfavorable 
performance.  A  well-documented  schedule  is  essential  for 
validating  and  defending  a  baseline  schedule,  analyzing 
changes  in  the  program  schedule,  and  identifying  the  reasons 
for  variances  between  estimates. 


Source:  GAO  analysis  of  Air  Force  data. 
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Appendix  II:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Cost  Estimate 


This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  Defense  Enterprise  Accounting  and  Management  System  (DEAMS) 
cost  estimate  supporting  the  February  2012  Milestone  B  decision1  met  the 
characteristics  of  a  high-quality  cost  estimate.  Table  4  provides  the 
detailed  results  of  our  analysis. 

GAO’s  methodology  includes  five  levels  of  compliance  with  its  best 
practices.2  “Not  met”  means  the  program  provided  no  evidence  that 
satisfies  any  of  the  criterion.  “Minimally  met”  means  the  program  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion.  “Partially  met” 
means  the  program  provided  evidence  that  satisfies  about  half  of  the 
criterion.  “Substantially  met”  means  the  program  provided  evidence  that 
satisfies  a  large  portion  of  the  criterion.  “Fully  met”  means  the  program 
provided  evidence  that  completely  satisfies  the  criterion. 


Table  4:  Assessment  of  DEAMS  Cost  Estimate  Compared  to  Best  Practices 


Characteristic 


Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Comprehensive  The  cost  estimate  includes  all  Fully  met 
life-cycle  costs.  The  cost 
estimate  completely  defines  the 
program,  reflects  the  current 
schedule,  and  is  technically 
reasonable.  The  cost  estimate 
work  breakdown  structure  is 
product  oriented,  traceable  to 
the  statement  of  work,  and  at  an 
appropriate  level  of  detail  to 
ensure  that  cost  elements  are 
neither  omitted  nor  double- 
counted.  The  estimate 
documents  all  cost-influencing 
ground  rules  and  assumptions. 


The  DEAMS  cost  estimate  included  both  government  and 
contractor  costs  of  the  program  over  its  life  cycle.  The 
documentation  provided  a  description  of  the  purpose  of  the 
estimate;  a  technical  description  of  the  program  including 
functional  relationships;  and  technical  risks  (e.g.,  hours  per 
test  activity  and  the  resolution  for  any  identified 
deficiencies). 

The  work  breakdown  structure  clearly  outlined  the  end 
product  and  major  work  to  be  performed.  The  program 
provided  supporting  documentation  that  showed  the  ground 
rules  and  assumptions,  such  as  the  total  number  of  users, 
use  of  a  commercial-off-the-shelf  product,  and  licensing 
requirements.  The  estimate  was  based  on  a  cost  element 
structure  as  stated  in  the  Department  of  Defense 
Automated  Information  System  Economic  Analysis  Guide. 


Milestone  B  is  considered  the  official  start  of  the  program  according  to  the  Defense 
Acquisition  Management  System  Framework.  The  framework  is  intended  to  translate 
mission  needs  and  requirements  into  stable,  affordable,  and  well-managed  acquisition 
programs. 

2GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs ,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 
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Appendix  II:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Cost  Estimate 


Characteristic  Best  practice 

Well-documented  The  documentation  should 

capture  the  source  data  used, 
the  reliability  of  the  data,  and 
how  the  data  were  normalized. 
The  documentation  describes  in 
sufficient  detail  the  calculations 
performed  and  the  estimating 
methodology  used  to  derive 
each  work  breakdown  structure 
element’s  cost.  The 
documentation  describes  step 
by  step  how  the  estimate  was 
developed  so  that  a  cost  analyst 
unfamiliar  with  the  program 
could  understand  what  was 
done  and  replicate  it.  The 
documentation  discusses  the 
technical  baseline  description 
and  the  data  in  the  baseline  are 
consistent  with  the  estimate. 

The  documentation  provides 
evidence  that  the  cost  estimate 
was  reviewed  and  accepted  by 
management. 


Assessment  Key  examples  of  rationale  for  assessment 

Substantially  met  The  documentation  identified  the  source  of  the  data  used 
and  that  the  data  were  normalized.  Major  cost  drivers  were 
also  identified,  such  as  personnel  costs,  support  costs,  and 
system  development.  However,  there  was  no  discussion  of 
efforts  taken,  if  any,  to  ensure  the  reliability  of  the  data 
used. 

The  documentation  identified  the  methodology  used  for 
each  cost  element  in  sufficient  detail  to  replicate  it.  The 
document  also  contained  the  rationale  for  selecting  the 
methodology.  Technical  descriptions  in  the  cost  model  were 
consistent  with  the  Cost  Analysis  Requirements  Description, 
which  contained  a  system  overview  outlining  the 
incremental  approach  being  used  to  develop  DEAMS. 

The  DEAMS  Program  Management  Office  presented 
evidence  of  receiving  approval  of  the  estimate  through 
briefings  to  management  although  there  was  no  reference 
in  the  briefing  to  the  risk  and  uncertainty  analysis  that  was 
performed. 


Accurate  The  cost  estimate  results  are  Substantially 

unbiased,  not  overly 
conservative  or  optimistic  and 
based  on  an  assessment  of 
most  likely  costs.  The  estimate 
has  been  adjusted  properly  for 
inflation.  The  estimate  contains 
few,  if  any,  minor  mistakes.  The 
cost  estimate  is  regularly 
updated  to  reflect  significant 
changes  in  the  program  so  that 
it  is  always  reflecting  current 
status.  The  estimate  is  based  on 
a  historical  record  of  cost 
estimating  and  actual 
experiences  from  other 
comparable  programs. 

Variances  between  planned  and 
actual  costs  are  documented, 
explained,  and  reviewed. 


met  The  program  provided  a  risk  and  uncertainty  analysis  that 
documented  the  range  of  all  possible  costs.  This  analysis 
showed  that  the  results  were  unbiased  and  reflected  the 
most  likely  costs.  The  DEAMS  cost  model  detailed  the 
calculations  and  inflation  indexes  underlying  the  estimated 
costs.  Calculations  within  the  model  can  be  traced  back  to 
supporting  documentation.  The  estimate  did  not  contain  any 
mistakes.  In  addition,  the  cost  model  was  updated  annually 
to  incorporate  actual  costs  expended  in  prior  fiscal  years. 
The  estimate  was  based  on  historical  data  as  appropriate. 

However,  the  cost  estimate  did  not  discuss  variances 
between  planned  and  actual  costs,  which  would  enable 
estimators  to  assess  how  well  they  are  estimating  program 
costs  and  identify  lessons  learned. 
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Appendix  II:  Assessment  of  the  Defense 
Enterprise  Accounting  and  Management 
System  Program’s  Cost  Estimate 


Characteristic  Best  practice 


Assessment  Key  examples  of  rationale  for  assessment 


Credible  A  risk  and  uncertainty  analysis  Substantially  met  The  program  management  office  provided  a  risk  and 

was  conducted  that  quantified  uncertainty  analysis  that  quantified  the  risks  for  the  major 

the  imperfectly  understood  risks  cost  drivers  (e.g.,  personnel,  deployment  software  and 

and  identified  the  effects  of  integrated  logistics  support)  based  on  changes  in  the 

changing  key  cost  driver  assumptions  (e.g.,  pending  negotiations  and  level  of 

assumptions  and  factors.  Major  knowledge  about  support  concepts).  DEAMS  program 

cost  elements  were  cross-  officials  stated  that  some  cross-checking  was  performed 

checked  to  see  whether  results  using  cost  models;  however,  the  results  of  this  analysis 

were  similar.  An  independent  were  not  documented. 


cost  estimate  was  conducted  by 
a  group  outside  the  acquiring 
organization  to  determine 
whether  other  estimating 
methods  produce  similar  results. 
The  cost  estimate  includes  a 
sensitivity  analysis  that  identifies 
a  range  of  possible  costs  based 
on  varying  major  assumptions, 


The  Air  Force  Cost  Analysis  Agency  developed  an 
independent  cost  estimate  that  was  reconciled  to  the 
program’s  cost  estimate. 

However,  a  sensitivity  analysis  was  not  completed  for  each 
of  the  major  cost  drivers.  As  a  result,  the  cost  estimator  will 
not  have  a  clear  understanding  of  how  each  major  cost 
driver  is  affected  by  a  change  in  a  single  assumption  and 
thus  which  scenario  most  affects  the  cost  estimate. 


parameters,  and  data  inputs. 


Source:  GAO  analysis  of  Air  Force  data. 
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Department  of  Defense 


ASSISTANT  SECRETARY  OF  DEFENSE 
30 1  5  DEFENSE  PENTAGON 
WASHINGTON,  DC  2030 1-30 15 


1  6  JAN  20'* 

Mr,  Asif  A.  Khan 
Director 

Financial  Management  Assurance 
U.S,  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Khan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  Government  Accounting  Office 
(GAO)  Draft  Report,  GAO-14-152,  “DoD  BUSINESS  SYSTEM  MODERNIZATION:  Air 
Force  Business  System  Schedule  and  Cost  Estimates”  dated  November  26,  2013.  The 
Department’s  comments  are  enclosed. 


Sincerely, 


Assistant  Secretary  of  Defense 
(Acquisition) 


Enclosure: 
As  stated 
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Appendix  III:  Comments  from  the  Department 
of  Defense 


GAO-14-152  (GAO  CODE  197120) 

"DOD  Business  Systems  Modernization: 

Air  Force  Business  System  Schedule  and  Cost  Estimates" 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION:  The  GAO  recommends  that  the  Secretary  of  the  Air  Force  direct  the 
Under  Secretary  of  the  Air  Force,  in  his  capacity  as  the  Chief  Management  Officer,  to  consider 
an  make  any  necessary  adjustments  to  the  DEAMS  cost  estimate  after  addressing  our  prior 
recommendation  to  adopt  scheduling  best  practices. 

DoD  RESPONSE:  Concur  with  the  GAO  recommendation. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  website  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  http://www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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